System and method for intelligent bandwidth allocation on multi-track multimedia communication systems

ABSTRACT

Disclosed are systems, methods, and non-transitory computer-readable media for bandwidth allocation in multi-track media communication that can include receiving a set of inbound media tracks; resolving track priority configuration for the set of media tracks; resolving media constraints that are at least partially derived from properties of the set of media tracks; producing bandwidth allocation configuration based on at least the track priority configuration and media constraints; and allocating bandwidth to outbound media tracks within a communication link to a client device wherein bandwidth of the outbound media tracks is allocated based on the bandwidth allocation configuration.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of Spanish Patent Application No.P201831009, filed on Oct. 17, 2018, which is incorporated in itsentirety by this reference.

TECHNICAL FIELD

An embodiment of the present subject matter relates generally to mediacommunication and, more specifically, to system and method forintelligent bandwidth allocation on multi-track multimedia communicationsystems.

BACKGROUND

Group conferencing for video or audio conferencing continues has seentremendous growth as a form of communication in recent years. Trends inglobally distributed teams, workforces, and business operations havedriven the need for high quality and reliable media conferencing acrossa large number of participants. There are a variety of conferencingtopologies that have been explored.

A mesh network topology can be used to interconnect each of theparticipants together. In some implementations, each participant sendstheir media to all of the other participants and receives media fromeach of the participants. Each participant will have a connection toeach other participant. This topology can have high upload and downloadnetworking bandwidth requirements and can be computationally expensive.

A mixing topology may employ a multiparty conferencing unit (MCU), wherea central computing system takes media from the participants, processesthe media thereby mixing a resulting media result and transmitting theresulting media to the participants. Computational and networkingrequirements of the MCU can be expensive.

Many modern real-time multimedia communication systems, such as the onesbased on WebRTC (Web Real-Time Communication) standards, use mediabundling techniques for communicating different media sources asseparated media tracks. A relevant example of this includes the use ofSFUs (Selective Forwarding Units) architected for multiparty videoconferencing services. In an SFU-based network topology, theparticipants upload their media stream to an SFU computing resourcefunctioning as a media routing machine, and the SFU sends back the mediastreams of the other participants. In those, every subscriber receivesthe rest of participant's videos as separate tracks that share the sametransport ICE (Interactive Connectivity Establishment) connection. Dueto this, at every subscriber's access link, these tracks compete for thesame bandwidth. That link may, however, have limited capabilities, whichcan result in various issues. One common approach used in today'ssystems is uniform allocation so that all tracks are assigned the verysame bandwidth (i.e. the available link bandwidth is divided by thenumber of video tracks being sent through that link). This may beaccompanied by several issues that can result in media tracks failing todeliver an appropriate media experience to an end user. Thus, there is aneed in the media communication field to create a new and useful systemand method for intelligent bandwidth allocation on multi-trackmultimedia communication systems. This invention provides such a new anduseful system and method.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, which are not necessarily drawn to scale, like numeralsmay describe similar components in different views. Like numerals havingdifferent letter suffixes may represent different instances of similarcomponents. Some embodiments are illustrated by way of example, and notlimitation, in the figures of the accompanying drawings in which:

FIG. 1 is a schematic representation of a system for intelligentbandwidth allocation;

FIG. 2 is a schematic representation of bandwidth management systemmanaging media track communication to multiple participants throughmultiple communication links.

FIGS. 3 and 4 are detailed block diagram representations of a system forintelligent bandwidth allocation.

FIG. 5 is a flowchart representation of a method for intelligentbandwidth allocation.

FIG. 6 is a schematic representation of a variation of resolving trackpriority.

FIG. 7 is a flowchart representation of one variation for producingbandwidth allocation configuration.

FIG. 8 is a block diagram illustrating components of a machine,according to some example embodiments, able to read instructions from amachine-readable medium (e.g., a machine-readable storage medium) andperform any one or more of the methodologies discussed herein.

FIG. 9 is a block diagram illustrating components of a machine,according to some example embodiments, able to read instructions from amachine-readable medium (e.g., a machine-readable storage medium) andperform any one or more of the methodologies discussed herein.

FIG. 10 is a system configuration for clients and a software platformcommunicating through a network.

DETAILED DESCRIPTION

In the following description, for purposes of explanation, variousdetails are set forth in order to provide a thorough understanding ofsome example embodiments. It will be apparent, however, to one skilledin the art, that the present subject matter may be practiced withoutthese specific details, or with slight alterations.

Reference in the specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiment is included in at least one embodimentof the present subject matter. Thus, the appearances of the phrase “inone embodiment” or “in an embodiment” appearing in various placesthroughout the specification are not necessarily all referring to thesame embodiment.

For purposes of explanation, specific configurations and details are setforth in order to provide a thorough understanding of the presentsubject matter. However, it will be apparent to one of ordinary skill inthe art that embodiments of the subject matter described may bepracticed without the specific details presented herein, or in variouscombinations, as described herein. Furthermore, well-known features maybe omitted or simplified in order not to obscure the describedembodiments. Various examples may be given throughout this description.These are merely descriptions of specific embodiments. The scope ormeaning of the claims is not limited to the examples given.

Disclosed are systems, methods, and non-transitory computer-readablemedia for intelligent bandwidth allocation in multiparty communications.

Overview

A system and method for intelligent bandwidth allocation on multi-trackmultimedia communication systems of an embodiment function to deliver anenhanced media experience that accounts for numerous use-cases, mediaformats, and bandwidth constraints. In particular, the system and methodcan be used in media routing to intelligently split bandwidth amongmultiple media tracks that share the same communication link such aswhen multiple video tracks are streamed to a participant in amulti-party video call as shown in FIG. 1. The system and method candynamically adjust bandwidth allocation based on a variety of factorssuch as application-specific prioritizes (e.g., use case of theapplication or settings/state of an end-user client), media limitations,bandwidth usage and limitations, and/or other factors.

The system and method preferably use a bandwidth management system indynamically adjusting and/or setting bandwidth allocation for a numberof tracks within at least one communication link. More preferably, thebandwidth management system dynamically adjusts and/or sets bandwidthallocation for media tracks sent over communications links to multipleparticipants. The dynamic adjustment may be individualized for selectparticipants in some variations as shown in FIG. 2. The system andmethod may be further extended such that management could be appliedacross multiple links for different multiparty conversations andpossibly serving different use-cases. Herein, the system and method areprimarily discussed in the context of a single link (i.e., multipartyconversation), which as one reasonably skilled in the art wouldappreciate may be extended to more than one link.

A bandwidth management system can be architected to assess manyconditions, which may be configured in execution as heuristic-basedprocessing of conditions, algorithmic/machine-learned processing of theconditions, or processed through any suitable detection/classificationapproach. The bandwidth management system preferably manages outboundcommunication links according to multiple factors. Three potentialfactors can include media track priority, media track limitations, andbandwidth usage and limitations. The various factors may be assessedsuch that a set of conditions is satisfied and a resulting assignment ofbandwidth usage and configuration for an outbound link determined.

In one exemplary implementation, a bandwidth management system can applymanagement processes that assign priorities, assign maximum and minimumbandwidth constraints, and assign bandwidth.

As a first process, tracks can be assigned priorities, which may dependon the track content, on the use-case, on the application or end-userpreferences, and/or other contextual properties. Track priorities may beautomatically determined through processing and analysis of the mediacontent. For example, an audio track can be processed for speech or avideo track may be processed for motion or human presence. In anothervariation, media track priorities can be set in response to applicationspecific configuration (potentially set for an entire account scope) ordetected end-user behavior on a receiving client device (e.g., browseror smartphone).

As a second process, tracks can be assigned maximum and minimumbandwidth constraints, which may depend on the track encoding, tracktransport, on the use-case or on the application, end-user preferences,and/or other constraint properties. Bandwidth or other media constraintsmay be obtained through analysis of the inbound media links uploaded byparticipants. Bandwidth constraints may additionally or alternatively beset by an associated developer account or in response to end user inputon a client device.

As a third process, tracks can be assigned bandwidth in an iterativeperiodic scheme through an optimization algorithm. Various conditions,targets, and/or guarantees can be specified through such a process. Thebandwidth assignment process preferably operates to satisfy a set ofproperties. In one potential property, tracks are assigned as muchbandwidth as possible provided that the other property conditions or aselect set of property conditions are/is satisfied. As another potentialproperty, the sum of consumed bandwidth by all the tracks must be lessthan or under the total link available bandwidth. As another potentialproperty, tracks are allocated a bandwidth under their respectivemaximum possible bandwidth. As another potential property, tracks areeach allocated a bandwidth over its minimum or else the track isswitched-off. As another potential property, track switch-off and -onfollows priorities so that lower priority tracks are always switched-offfirst and switched-on last in relation to on/off state of other tracks.As another potential property, track bandwidth allocation can follow amonotonic function of their priorities. For example, for the identityfunction, that means that tracks with double priority are assigneddouble bandwidth as long as they are within their constraint bounds.

As a complimentary process, the bandwidth management system can receiveactual bandwidth data associated with a communication link and thatserves as an input to the bandwidth assignment process. Other forms ofusage feedback may additionally be received such as client media usage,which can include end-user prioritization (e.g., which media tracks arebeing consumed and how) or media limits (e.g., client device orapplication limitations).

The bandwidth management system is preferably configurable and cansupport a programmatic interface and preferably an applicationprogramming interface (API) or other suitable mechanism forprogrammatically setting properties used in managing bandwidth.

The system and method can be used in a variety of media communicationuse-cases. Preferably, the system and method are used in connection withmulti-party communication. More specifically, the system and method areused with a multi-party video chat or call communication session. Thesystem and method may additionally or alternatively be used with anycommunication or synchronous data stream. Media streams can be anysuitable media/data transmission of data to facilitate synchronouscommunication through one or more mediums. The system and method may beused for facilitating video, voice/audio, multimedia (e.g.,screensharing or game/interactive media), data, and/or other suitableforms of media streams. The system and method can be used to improveconference calls where there are many active or potentially activeparticipants, one (or a few) to many communications (e.g., talks with asmall number of active participants and many observers), and/or othercommunication scenarios.

Herein, references to a media track are used to characterize a mediastream, which is a data transmission of a synchronous data set. A mediatrack will generally originate from one participant. Though, in somevariations, a media track processed and used by the system and methodmay have been a new media stream generated from media streams frommultiple other sources. A media track can be a streamed video and/oraudio of a participant collected at a client device (e.g., a cameraand/or microphone on a computing device like a phone or computer). Amedia track may alternatively be any suitable real-time data stream asmentioned above.

Herein, references to a communication link characterize a synchronousdata connection from the system to a client device. Each client device(and accordingly each active participant sending or receivingmedia/data) will preferably have an established communication link. Insome preferred implementations, the communication link is facilitatedover WebRTC (Web Real-Time Communication), which is a technology adoptedby many web browsers, mobile applications to provide a common set ofprotocols and conventions in facilitating real-time communications. InWebRTC, the communication link can be established as an“RTCPeerConnection”. One or more media tracks can be transmitted to aconnected client device over the communication link.

The media tracks are preferably communicated through a communicationlink to the participants of a communication session. Herein, referenceto a communication session is used to characterize the groupcommunication of a set of participants, which can involve the collectivedata communication used in enabling communication between the set ofparticipants. In general, one communication link is preferablyestablished for each participant. Though alternative variations mayenable end participants to at least partially share a communicationlink. For example, one communication link may be communicated to anintermediary proxy or router device that retransmits the media tracksover two different communication links. In this example, the outboundcommunication link from the system to the router may be, from theperspective of the system, shared by two participants.

In one implementation, the system and method may be implemented inconnection with a media communication service such as a conference callservice wherein people use the service explicitly for facilitatingvideo, media, data and/or other forms of synchronous communications. Aconference call service can generally be implemented through acloud-based/server-based computing infrastructure, which may be hostedin a single location or distributed across multiple locations. Thecomputing infrastructure can include servers, data storage, network andvirtualization software, and/or any suitable computing systems. Aconference call service may include an account system, whereinindividual accounts can setup and configure individualized use of theconference call service. Within the scope of an account, multipartyconference calls may be conducted or scheduled. The configuration andusage of the conferencing capabilities of the conference call service byan account can serve as input to the bandwidth management system inenhancing the bandwidth management for improved performance based onproperties of the conference calls and/or usage patterns of conferencesof a given account.

In an alternative implementation, the system and method may beimplemented in connection with a media communication developer tool orservice wherein a communication service facilitates communicationsessions on behalf of another service. For example, a multitenantcommunication-as-a-service (CaaS) computing platform may use the systemand method for a video conferencing tool used by various applications orservices to enable video conferencing within their own application orservice. As one potential feature, the system and method may enable auser of the CaaS platform to specify a user-case or layout from a set ofpre-configured modes. Exemplary modes could include grid-video chat,presentation, and collaboration.

A multitenant CaaS computing platform can also be implemented throughcloud-based/server-based computing infrastructure, which may be hostedin a single location or distributed across multiple locations. Thecomputing infrastructure can include servers, data storage, network andvirtualization software, and/or any suitable computing systems. A CaaScomputing platform may include an account system, wherein individualaccounts can setup and configure individualized use of the communicationservices offered by the CaaS computing platform. In this case, theaccount holders will generally be developer accounts or managingaccounts and used in developing and integrating communication service ofthe CaaS computing platform with an outside communication service orapplication. As such, the CaaS may be used in a wide variety ofcommunication scenarios. The system and method can be employed inaddressing the various needs and use-cases of a diverse set ofdevelopers. For example, one account may make use of video or audioconferencing service of the CaaS computing platform in providing verylarge video conferences with 10's, hundreds, or even thousands ofparticipants, and another account may integrate video conferencingservices of the CaaS computing platform for collaboration of 2-8participants. Many other forms of use may be created using the CaaScomputing platform, and the system and method can facilitate dynamicallyadapting to the usage conditions.

As a conferencing call service operating as a conferencing computerplatform and/or as a CaaS computing platform, the system and method canpreferably offer flexibility and individualized control over the optionsin how media is communicated. While many legacy solutions only had toconsider multiparty conferencing for one application and/or use-case,the system and method can preferably accommodate a wide variety ofrequirements and preferences of the outside applications or softwareservices using the system and method.

In another implementation, the system and method may be implemented as adynamic SFU computing device. In such an implementation, an SFUcomputing device which can be a computer, a server, router, a virtualmachine operating on a computer, and/or any suitable application orsoftware package or hardware-implemented system that includes amachine-readable storage medium comprising instructions that, whenexecuted by one or more processors of a machine, cause the machine toperform operations of the system and method.

The system may additionally or alternatively be implemented within adynamic SFU system, which may include a plurality of dynamic SFUcomputing devices that collectively operate. A plurality of dynamic SFUcomputing devices may individually handle multiple multi-partyconferencing sessions A plurality of dynamic SFU computing devices mayalternatively cooperatively handle a conferencing session wherein eachdynamic SFU facilitates at least a portion of the processes used inintelligently management the communication link.

As one potential benefit, the system and method can manage media tracksin a use-case relevant manner. This may result in improved userexperience for participants involved in a communication. The system andmethod preferably provide a mechanism for assigning priorities to thedifferent video tracks and such priorities should influence thebandwidth allocation so that higher priority tracks should be assignedmore bandwidth than lower priority tracks.

As another potential benefit, the system and method can accommodatemedia track bandwidth limitations in coordination with applicationspecific preferences. For example, there are video codecs and codingtechniques that can depend on a video track having a minimum bandwidthunder which the video track cannot be sent. There are also situations inwhich tracks have a maximum bandwidth and any extra capability allocatedto them would be lost. The system and method preferably incorporatessuch constraints into a dynamic bandwidth allocation process.

As another potential benefit, the system and method function toaccommodate a variety of scenarios. The priorities of media tracks andtheir coordination can be specifically configured for an application oreven a particular communication session. In some variations, thisprioritization can change within a communication session.

As another potential benefit, the system and method may provide amechanism for outside configuration and control. The system and method,in some variations, can provide a programmatic interface such as an APIor some network communication protocol for relaying information. Theprogrammatic interface may be used in specifying application,conference, or end-user specific parameters that may be used inadjusting the bandwidth allocation.

As another potential benefit, the system and method may dynamicallyrespond to usage by an end client device in a conference or usagepatterns of conferences associated with an account (e.g., a developer,host, or participant account). The system and method may not onlyprovide some mechanism for directed control, but the system and methodmay additionally automatically adjust bandwidth allocation in anintelligent manner that is context sensitive. In some variations, thesystem and method may employ learning applied per client device, useraccount of a participant, application or developer account, or acrossmultiple entities in a platform (e.g., platform-wide learning).

System

As shown in FIG. 3, a system for intelligent bandwidth allocation of amulti-track multimedia communication of a preferred embodimentpreferably includes a media track interface 100 through which multipleinbound media tracks are received, a policy manager 200 through whichapplication dependent configurations can be specified, a bandwidthmanagement system 300 that can dynamically adjust bandwidth allocationof a set of media tracks when establishing a communication link to oneor more participant client devices. A set of media tracks is transmittedto participant client device through the communication link. Thebandwidth management system 300 preferably includes a track prioritymanager 310, a media bandwidth manager 320, a bandwidth estimator 330,and a bandwidth assignment manager 340 as shown in FIG. 3.

The system is preferably implemented in a server or server system thatis a network accessible device or system (e.g., a collection computingresources). The system is generally remote from the participant clientdevices in that the participants transmit communications to the system.In some variations, described below, the system may additionally includeclient device service software that can operate on a client device,where the client device may provide bandwidth feedback, participantmedia priority, client device media constraints and/or other forms offeedback and input.

The media track interface 100 functions as an interface for inboundmedia tracks (e.g., media streams). Inbound media tracks may also bereceived through the communication link, wherein the inbound mediatracks are uploaded from a participant client device to the system. Thesystem preferably operates on a number of inbound media tracks intendedfor one or more subscribing client devices. A media track preferablyrefers to audio and/or video tracks being generated by remotepublishers. Media tracks are typically encoded using a codec that may ormay not be a SVC (Scalable Video Codec) or may or may not be representedby different independent sub-tracks (e.g. Simulcast). The media tracksmay additionally or alternatively include other suitable types of mediatracks such as a data stream for interactive multi-media applications.The media track interface 100 may have media tracks directlycommunicated from a publishing client device. The media track interface100 may alternatively receive media tracks redirected from some othermedia/communication system. The media track interface 100 preferablyacts as an input to the bandwidth management system 300.

The policy manager 200 functions to manage policy configuration usablein one or more communication links and/or communication sessions. Thepolicy manager 200 preferably stores and communicates storedconfiguration or policy to the bandwidth management system 300. A policymanager 200 can enable application-dependent configuration to be definedfor individual communication links, communication sessions, groups ofcommunication sessions, communication sessions of an account,communication sessions of a sub-account, and/or for any suitable scope.Application-dependent configurations preferably characterize anyinformation provided by a RTC (Real Time multimedia Communication)application determining how the bandwidth management system 300 behaves.This information may include publish/subscribe information, qualitycontrol information, media processing information, bandwidth properties,media dimension properties, track count limits, and/or other factors.

In one variation, there may be preset configuration properties that canbe conveniently enabled by setting a communication session to aparticular mode. Exemplary modes could include grid-video chat,presentation, and collaboration. In a grid-video chat, the media of thelinks will generally be presented in a substantially uniform manner. Ina presentation mode, there will generally be a dominant link that isacting as the main participant. In some cases, that presenter may besharing a screen. The other participants may or may not be activelyparticipating. For example, the policy manager 200 could support abroadcast variation of the presenter mode wherein a select set ofparticipants can be indicated as the primary broadcasters and theremaining participants as “view” only. In a collaboration mode, thepriority of a link changes based on participation. Participation may bespeaker based, movement based, or based on any suitable metric ofparticipation. As an example, the policy manager 200 could support acollaboration conference mode that enables settings in theapplication-dependent to adjust settings in a manner suitable for asmall conference call with dynamic participation by the involvedparties.

The policy manager 200 may set media policy, which may function todetermine the relevance of a track in the context of a specificuse-case. Media policies preferably involve some kind of media analysis.Examples of media policies may include speaker policy mode, movementactivity policy mode, presence activity policy mode, and/or othersuitable types of policies or combination of such policies.

A speaker policy mode can set the priority of the media tracks of aspecific publishing source (e.g. participant) based at least in part onthe participant's speaking activity. This could be determined throughaudio analysis and/or video analysis. Media track priority may beadjusted based on real-time speaker detection, history of speaking, andproperties of the speech. For example the tone, content, volume, and/orother factors related to speech detection can be used.

A movement activity policy mode can set the priority of the media tracksof a specific publishing source based at least in part on the presence,degree, and possibly type of movement on the source's video track orother type of media track. This may be used to change priority of mediatracks in response to visual activity in a video track.

A presence activity policy mode can set the priority of the media tracksof a specific publishing source based at least in part on the presenceof a specific object or property on the media tracks (e.g. presence of aface, presence of a car, etc.). This may be used to depend on thepresence of a human in a human video stream. In a related example,policy may depend on various factors related to the presence of thedetected object(s). For example, the size of a human face can be afactor. In some use-cases, presence activity policy may deprioritizevideo tracks with people far from the camera.

The policy manager 200 will preferably impact policy dependent trackpriorities. Track priorities preferably indicate the priorities of thedifferent media tracks as established by the applied policies.

In one implementation, speaker activity, movement activity, presenceactivity, and/or other factors may be determined by the system throughprocessing and analyzing a media track. This analysis can determine oneor more properties on which a policy mode may be defined. Track analysismay be facilitated by a media manager 322 or any suitable systemcomponent. In another variation, media track factors may be communicatedthrough a data signaling communication transmitted by a client device oran outside computing device and received at the system (e.g., the policymanager 200).

Policy records can be preset or preconfigured so that they are enabledwhen establishing the communication link. Policy records mayalternatively be set, changed, removed, or otherwise mutated during acommunication session.

The policy manager 200 preferably includes a policy data storage system(e.g., a database system) used to store and maintain records of thepolicy. Policy records, which are machine-storable data records definingpolicy properties and rules, may be stored and associated with anaccount, a sub-account, communication session, a participant, or withany suitable scope.

Additionally or alternatively, the policy manager 200 may include orinterface with an application programming interface (API) or anothersuitable programmatic interface. An API is preferably used by outsideparties in specifying policy.

The application programming interface (API) service 210 functions as aprogrammatic mechanism through which external entities can manageaccount settings, communication sessions, participants, individualcommunication links, and/or other aspects.

An API service is preferably a RESTful API but may alternatively be anysuitable API such as SOAP, GraphQL, or custom protocol. The RESTful APIworks according to an application layer request and response model. Anapplication layer request and response model may use an HTTP-basedprotocol (HTTP or HTTPS), SPDY, or any suitable application layerprotocol. Herein, HTTP may be used, but should not be interpreted asbeing limited to the HTTP protocol. HTTP requests (or any suitablerequest communication) to the communication platform preferably observethe principles of a RESTful design. RESTful is understood in thisdocument to describe a Representational State Transfer architecture asis known in the art. The RESTful HTTP requests are preferably stateless,thus each message communicated contains all necessary information forprocessing the request and generating a response. The API service caninclude various resources, which act as API endpoints that can act as amechanism for specifying requested information or requesting particularactions. The resources can be expressed as URI's or resource paths. TheRESTful API resources can additionally be responsive to different typesof HTTP methods such as GET, Put, POST and/or DELETE. Information abouta conversation is preferably stored and made accessible throughconversation API resources. Alternative types of APIs may use differentapproaches for organization programmatic interactions with thecommunication system 100.

The bandwidth management system 300 functions to set or determinedynamic bandwidth for a communication link. The bandwidth managementsystem 300 can function as an intelligent dynamic selective forwardingunit that manages, clones, routes and processes media tracks based on anumber of application dependent configurations. The bandwidth managementsystem 300 may be implemented within a single computing device (e.g., aserver) or virtual machine. The bandwidth management system 300 mayalternatively comprise of multiple computing resources including variousforms of hardware, software, and/or virtualized resources. The bandwidthmanagement system 300 can preferably be applied to each individualcommunication link. In some variations, the bandwidth management system300 may coordinate bandwidth management ad routing of media tracksacross communication links across multiple communication links of thesame communication session. Additionally, the bandwidth managementsystem 300 may coordinate bandwidth management and routing of mediatracks across communication links of one or more communication links fordifferent communication sessions in parallel.

The bandwidth management system 300 preferably sets bandwidth allocationsettings based on a variety of settings including application-dependentconfiguration, media limitations, bandwidth availability, and/or otherfactors. As shown in FIG. 3, one preferred variation of the bandwidthmanagement system 300 includes: a track priority manager 310, which setspriority of media tracks based on various factors; a media bandwidthmanager 320, which sets media track related constraints used indetermining bandwidth allocation; a bandwidth assignment manager 340,which applies an algorithm to assess track priority, constraints,bandwidth, and/or other properties in setting bandwidth allocation for aset of media links; and a bandwidth allocator 350. The bandwidthmanagement system 300 may additionally include a bandwidth estimator 330which can predict or supply bandwidth data used by the bandwidthassignment manager 340 when setting bandwidth allocation.

The track priority manager 310 functions to account for policy settings.The track priority manager 310 can be communicatively connected to thepolicy manager 200, wherein the policy manager 200 preferably transmitsor communicates policy information to the track priority manager 310,and the track priority manager 310 includes machine configurationconfigured to implement the policy and set priority of the media tracks.In one variation, the track priority manager 310 includes a media policymanager 312 and an application priority manager 314 as shown in FIG. 4.The media policy manager 312 preferably generates track priorities basedon media analysis. The application priority manager 314 preferablyprocesses the track priorities along with the application-dependentconfiguration to generate effective track priorities. The effectivepriorities are preferably used as the output of the track prioritymanager 310 and processed by the bandwidth assignment manager 340.

The media policy manager 312 functions to adjust track priority based onproperties and/or content of the media track. The media policy manager312 preferably applies one or multiple media policies to set therelevance of a track in the context of a specific use-case. The mediapolicy manager 312 preferably includes or connects to a media analysiscomponent. As described above, exemplary media-related policies caninclude a speaker policy mode, a movement activity policy mode, and/or apresence activity policy mode. Policy configuration can include property(or properties) set to activate or trigger activation of one or more ofthese modes.

In a speaker policy mode, the media policy manager 312 is configured toset the priority of the media tracks of a specific publishing source(e.g. participant) based at least in part on the participant's speakingactivity. This could be determined through audio analysis and/or videoanalysis. Media track priority may be adjusted based on real-timespeaker detection, history of speaking, and properties of the speech.For example the tone, content, volume, and/or other factors related tospeech detection can be used.

In a movement activity policy mode, the media policy manager 312 isconfigured to set the priority of the media tracks of a specificpublishing source based at least in part on the presence, degree, andpossibly type of movement on the source's video track or other type ofmedia track. This may be used to change priority of media tracks inresponse to visual activity in a video track.

In a presence activity policy mode, the media policy manager 312 isconfigured to set the priority of the media tracks of a specificpublishing source based at least in part on the presence of a specificobject or property on the media tracks (e.g. presence of a face,presence of a car, etc.). This may be used to depend on the presence ofa human in a human video stream. In a related example, policy may dependon various factors related to the presence of the detected object(s).For example, the size of a human face can be a factor. In someuse-cases, presence activity policy may deprioritize video tracks withpeople far from the camera.

The media policy manager 312 is preferably responsive to policy that isapplication dependent. In some variations, communication sessions ofeach account and/or each communication session may have customizedmedia-property related policy configured, which is applied by the mediapolicy manager 312.

The output of the media policy manager 312 is preferably a set ofmedia-related priorities. In one implementation priorities may becharacterized by ordering the media tracks from highest priority tolowest priority. In anther implementation, a priority rating can beassigned to each inbound media track.

The application priority manager 314 functions to set track prioritybased in part on application or use-case specific conditions. Thepolicies used in setting the priorities in the application prioritymanager 314 are based on outside factors. The application prioritymanager 314 preferably applies one or multiple contextual policies toset the relevance of a track.

Information provided by the application or an end-user/participantclient device may indicate or be used in setting the priority of thedifferent media tracks. This may be used so that unique usage andcontext of a specific application or use-case can be accounted for.

In one variation, the application priority manager 314 may directly setthe priority of one or more tracks based on external input. For example,a client device viewing a video track at a greater size may result inthat video track being set with the highest priority when sending in acommunication link to that particular client device. In another example,an application that has a set of pre-defined participants as having moreimportance (e.g., moderators of a conference call) than another set ofparticipants can have the high priority participants' priority set bythe application priority manager 314. In some variations, theapplication priority may act as an override of media-related priority.

In another variation, the application priority manager 314 may indicateprioritization of one or more tracks and this may be used in adjustingthe priority used in managing bandwidth. This may enable the relevantpriority of tracks to balance media-related priority andapplication-directed priority.

The output of the application priority manager 314 may provide a secondset of application-based track priority properties. In one variation,the application-based track priority properties may be in addition to aset of media-related priorities. In another variation, a generalized setof track priorities may be generated by assessing application-basedtrack priorities and media-related priorities.

The media bandwidth manager 320 functions to account for mediafunctionality and limitations. Different forms of media may havedifferent bandwidth dependent options, and the media bandwidth manager320 preferably incorporates such limitations and preferences. The outputof the media bandwidth manager 320 is preferably a set of constraintsfor the set of media links. The constraints may include a bandwidthminimum and/or maximum for one or more media tracks. Bandwidthconstraints may be set based on the media format, media type, codec, orother media-related property. A bandwidth constraint may alternativelybe set based on application-specific prioritizes (e.g., use case of theapplication or settings/state of an end-user client).

A bandwidth minimum will generally be set at a level below which themedia may not be adequately rendered by a client device. A bandwidthmaximum may be set at a threshold above which the incrementalimprovements in quality are diminished or have no significant differenceto the media link. The media constraints may be based on media-relatedproperties and/or application-dependent properties.

In one variation, the media bandwidth manager 320 includes a mediamanager 322 and a bandwidth constraint manager 324 as shown in FIG. 4.

The media manager 322 functions to provide media-related constraints.Media-related constraints may be inherent track related constraints.Media-related constraints preferably include the minimum track bandwidthand maximum track bandwidth. The minimum track bandwidth and maximumtrack bandwidth for each track may be determined by analyzing the mediaformat, media type, codec, or other media-related properties. As oneexample, in an SVC encoded video track, the minimum track bandwidth maybe the bandwidth of the lowest quality layer that can be extracted fromthe inbound SVC track. The media-related maximum track bandwidth is thethreshold at which allocating more than the maximum bandwidth to therelevant track would not make the track consume more bandwidth and/oralter the media quality.

The media manager 322 preferably generates inherent track constraintsbased in part on the media links. In some variations, the media manager322 may additionally include configuration to perform the media cloning,routing and processing required by SFU logic when the system is beingused as a dynamic SFU.

The bandwidth constraint manager 324 functions to incorporateapplication-dependent configuration in setting media link constraints.The bandwidth constraint manager 324 preferably takes in the inherenttrack constraints and application-dependent configuration to determineresulting track constraints.

In a preferred variation of the system, the bandwidth management system300 includes a bandwidth estimator 330, which functions to estimateeffective available bandwidth within a link. The bandwidth estimator 330preferably takes in bandwidth estimation feedback reported back from thesubscribers, reported through the communication links, or received fromany suitable source. The bandwidth estimation feedback is preferablyinformation regarding the constrained bandwidth link status that makesit possible to estimate the available bandwidth link.

In one implementation, the bandwidth estimator 330 operates incoordination with a client device service. The client device service isa software or hardware implemented component that transmits bandwidthinformation directly or indirectly to the bandwidth estimator 330 or toanother component of the system. The client device service can beimplemented through an application, SDK (software development kit),library, or other software service. Alternatively, client devicesinterfacing with the system may voluntarily comply with a protocol andprovide bandwidth estimations.

The client device services may additionally provide other client deviceinput such as end user media track usage and interactions, which can beused in updating the policy manager 200, the track priority manager 310and/or the media bandwidth manager 320.

The bandwidth assignment manager 340 functions as a form of optimizerthat assigns bandwidth allocation settings to be put into place by thebandwidth allocator 350. The bandwidth assignment manager 340 preferablytakes in track priorities from the track priority manager 310, trackconstraints from the media bandwidth manager, and bandwidth availabilityfrom the bandwidth estimator 330. The bandwidth assignment manager 340may additionally take in track status and track consumed bandwidthinformation, which may be reported from the media manager or anysuitable system. The bandwidth assignment manager 340 preferably outputsbandwidth allocation settings. The bandwidth allocation settings arepreferably a data model representing the settings for allocatingbandwidth when sent to one or more participants. The bandwidthassignment manager 340 can preferably coordinate multiple factors toappropriately act on the various constraints.

The bandwidth assignment manager 340 and more generally the bandwidthmanagement system 300 are preferably configured to evaluate the mediatrack factors (e.g., media track priority, constraints, and availablebandwidth) in accordance with a set of conditions. Those conditions maydefine a set of rules that define how bandwidth is allocated.

In one implementation, the bandwidth assignment manager 340 can includemachine instructions that when executed are configured to: maximize thesum of bandwidth consumed by all media tracks; verify a given link doesnot consumer more than the available bandwidth; verify that no tracksare over their respective maximum; verify all tracks are over theirminimum or are switched off; activate a switched off track to beswitched on if all the higher priority tracks have been allocated theirrequired maximum during a certain time; and regulate bandwidthallocation such that bandwidth allocation follows priorities. Additionalor alternative sets of conditions may alternatively be configured in thebandwidth assignment manager 340.

The bandwidth allocator 350 functions to apply determined bandwidthsettings to media tracks to a subscriber. The bandwidth allocator 350preferably takes one or more subscribed tracks and communicates thosetracks through a constrained bandwidth link to one or more specifiedsubscribers with adjusted outbound track bandwidth settings. Thebandwidth allocator 350 preferably takes the bandwidth allocationsettings of the bandwidth management system 300 and the media tracks andmodifies the media tracks to the bandwidth allocation settings. This caninclude setting minimum bandwidth, maximum bandwidth, enabling/disablingvideo, enabling/disabling audio, and/or making any suitable change. Aconstrained bandwidth link preferably refers to a packet-switchednetwork connection (e.g. an IP connection) where a number of outboundtracks are transported towards a subscriber. The network link underlyingthis connection may have constrained (i.e. limited) capabilities interms of bandwidth, jitter, packet loss, and/or other constraints.Outbound tracks refer to the tracks bundled and transported through theconstrained bandwidth connection and link towards its destination. Asubscriber refers to a client-side application, service, or deviceusually under the control of an application end-user, where the mediatracks are rendered.

Method

As shown in FIG. 5, a method for intelligent bandwidth allocation onmulti-track multimedia communication systems of an embodiment caninclude receiving a set of inbound media tracks S100, assigningbandwidth allocation in coordination with communication conditions S200,and allocating bandwidth to outbound media tracks of a communicationlink to a client device S300. Assigning bandwidth allocation preferablyadditionally includes resolving media track priority configuration forthe set of inbound media tracks S210, resolving media constraints forthe set of inbound media tracks S220, producing bandwidth allocationconfiguration based on at least the media track priority configurationand media constraints S240. The method may additionally includereceiving bandwidth estimation feedback and estimating availablebandwidth of the outbound link S230, which can be integrated withproducing bandwidth allocation configuration in S240. The method ispreferably implemented by a system such as the one described above, butany suitable system may alternatively be used.

Block S110, which includes receiving a set of inbound media tracks,functions to collect or otherwise obtain inbound tracks intended for atleast one subscriber. There is generally a plurality of inbound mediatracks, which may originate from various participants. The source of amedia track may also be a subscriber. Preferably, the various sources ofmedia tracks direct the media tracks to a central service such as somecommunication platform, where the intelligent bandwidth allocation canbe applied before sending to a subscribing participant. In the case of avideo conferencing tool, the various participants that have audio and/orvideo enabled upload a media track (e.g., video and/or audio stream) toa system with the allocation service.

In one variation, the inbound media tracks are preferably video mediatracks from a set of participants. In another variation, the inboundmedia tracks are audio media tracks. As discussed herein, the mediatracks may be a variety of types of media tracks. The type of mediatracks may be substantially across the participants (e.g., all audio),but in some instances or variations, the media tracks can be from a setof possible types of media tracks (e.g., video, audio, screencast,etc.).

In one variation, receiving an inbound media track is achieved over acommunication link between the bandwidth management system (or morespecifically a media track interface of the system) and a participantclient device. The communication link may additionally be used intransmitting the outbound media tracks to be delivered to participant.In a WebRTC implementation, the inbound media track can be a media trackuploaded over an RTCPeerConnection.

Block S200, which includes assigning bandwidth allocation incoordination with communication conditions, functions to process a setof factors related to the media link, the communication link, and thecurrent usage context. Assigning bandwidth allocation preferablydetermines per-track communication settings for transmission to a clientdevice of a participant based on a number of conditions. Assigningbandwidth allocation preferably assigns the bandwidth allocationaccording to application-dependent configuration, media trackconstraints, and the available bandwidth for a subscriber.Application-dependent policy can be used in combination to determine thebandwidth allocation intended for tracks of a link. Additionally oralternatively, real-time conditions of a client device of a participantand/or application context can be measured and used in changing mediabandwidth allocation.

As mentioned above, assigning bandwidth allocation in coordination withcommunication conditions S120 can additionally include resolving mediatrack priority configuration for the set of inbound media tracks S210,resolving media constraints for the set of inbound media tracks S220,and producing bandwidth allocation configuration based on at least themedia track priority configuration and media constraints S240. Morepreferably, assigning bandwidth allocation includes resolving mediatrack priority configuration for the set of inbound media tracks S210,resolving media constraints for the set of inbound media tracks S220,receiving bandwidth estimation feedback and estimating availablebandwidth of the outbound link S230, and producing bandwidth allocationconfiguration based on at least the media track priority configurationand media constraints S240.

Block S210, which includes resolving media track priority configurationfor the set of inbound media tracks, functions to set track prioritypreferences. The priority setting of the set of media tracks can laterbe used in assessing which media tracks to prioritize when allocatingbandwidth. Higher priority will generally be used to indicate that themedia track is deserving of greater bandwidth allocation while lowerpriority will generally indicate lower incentive for bandwidthallocation. The track priority settings can be based at least in part onsettings provided by an external entity or system. The priority may beapplication-dependent, which may mean the priority changes in responseto media of an application-related conversation and/or may mean thepriority changes in response to external indicators of priority from amanaging account or a signal received from a client device.

Application dependent track priorities can be customized for theparticular use-case to set priorities that are appropriate for thatuse-case. In a CaaS computing platform, it can be highly beneficial toprovide flexibility for an application using the services to customizeperformance for their use-case. Priorities may be based onassociated-participants (e.g., the origin of a media track), activity inthe media (e.g., speaker detection, motion detection, person/objectpresence, etc.), and/or any suitable media-track property. Applicationdependent track priorities can additionally be altered at any suitabletime. Alternatively, the track priority configuration can be specifiedwhen establishing the communication link.

In one variation policy can be used in part in determining media trackpriority configuration. The method may include receiving and settingpolicy configuration and resolving media track priority configurationbased in part on the policy. Policy may explicitly set media trackpriority for one or more media tracks or set configuration used indirecting modes of priority assessment. Policy can be set throughaccount settings, wherein the same priority is applied to multiplecommunication links associated with the account. In another variation,policy may be set for a particular communication session (e.g., multiplelinks involved in a shared communication or application session). Policycould additionally or alternatively be set per subscriber. Policy couldadditionally be set at different scopes and policy interpretation canappropriately determine an effective policy based on variouspolicy-overwriting rules.

Application-dependent policy may be used to set priority configurationto specify publish/subscribe information, quality control information,media processing information, track priorities, participant priorities,and/or other factors. The policy may additionally include media relatedpolicy such as various forms of policy dependent on media analysis suchas speaker detection, presence detection, movement detection, and thelike.

In one preferred variation, resolving media track priority configurationpreferably includes setting media-based track priority S212 and/orsetting use-case track priority S214 as shown in FIG. 6.

Block S212, which includes setting media-based track priority, whichfunctions to customize the priority of a media track based on propertiesand conditions of the media. Setting track media based priority mayinclude processing and/or analyzing the media tracks and setting amedia-based track priority for each media track. Processing and/oranalyzing the media track can include performing computer vision forvideo or image processing of a video or image based media track.Processing and/or analyzing the media track can include performingspeech recognition, speech analysis, or other forms of audio analysisfor a media track with audio media.

In one variation, the method may include setting media-based trackpriority based on at least one policy mode. One or more policy mode maybe configured or selected from a set of modes that can include aspeaking activity mode (e.g., speaker detection), a movement activitymode (e.g., movement detection), and presence activity mode (e.g.,presence detection).

In a speaker policy mode, resolving media track priority can includeprocessing audio of the media track and setting priority in response toaudio analysis. Processing of audio is preferably performed for at leasta subset of the media tracks and preferably all media tracks in manycases. Processing audio may include detecting the presence of sounds,detecting spoken words (e.g., to prioritize), detecting noise ornon-spoken audio (e.g., to deprioritize) converting speech to text fornatural language processing, detecting and analyze the content of thespeech, analyzing sentiment of audio or otherwise classifying tone ofaudio, and/or performing any suitable form of audio analysis. Processingof audio is preferably applied in detecting speaking activity in a mediatracks and rating level of speaking activity relative to other mediatracks. The results of audio analysis are preferably used in settingpriority of a media track. Media track priority may be adjusted based onreal-time speaker detection, history of speaking, and properties of thespeech. For example the tone, content, volume, and/or other factorsrelated to speech detection can be used.

In a movement activity policy mode, resolving media track priority caninclude analyzing movement in video content of a media track and settingpriority in response to the video analysis. Analyzing movement mayinclude performing computer vision or other suitable video/imageprocessing techniques in determining presence of movement, detectinghuman motion, classifying type of movement, and/or otherwise analyzingthe visual activity in video content of a media track. The movementactivity and more generally the visual activity detected in a mediatrack may be used in setting priority of a media track.

In a presence activity mode, resolving media track priority can includeanalyzing object presence in a media track and setting priority inresponse to object presence. Presence activity can be a variation ofvideo analysis and will preferably include performing computer vision orother suitable video/image processing techniques in detecting presenceof a specific object or a type of object presence in the video contentof a media track. Object detection can be used in detecting presence ofa face. Video/image processing may further be used in characterizing orproviding analysis of the properties of the detected object such asdetecting and characterizing human attention (e.g., direction ofattention), detecting speech visually (e.g., detecting mouth movement),and other suitable forms of presence detection. Object presence and inparticular human presence may be used in setting priority of a mediatrack.

Block S214, which includes setting use-case track priority, functions toadjust priority in response to various application-dependent factors.Use-case track priority can be set in response to an outside indicator.Setting of use-case track priority can be set in response to state ofpreconfigured policy, detected feedback from a client device, aprogrammatic update to policy from a managing account, and/or otherfactors. Use-case track priority in one variation can be explicitsetting of priority where the priority of one or more media tracks aredirectly set, which may override media-based priority or otherprioritization. Use-case track priority in another variation mayalternatively weight or apply a modification to priority of a mediatrack, which may emphasize or deemphasize a priority setting frommedia-based priority. For example, a key participant may have priorityweighted so that any activity by the key participant is given greaterpriority to less involved participants.

In one preferred approach, setting output media track priority caninclude setting an initial priority based on media-based priority ofeach media track and overriding the media-based priority with use-casepriority for individual media tracks where an explicit use-case priorityis set.

Priority configuration may additionally be specified by indication of apre-configured mode. A pre-configured mode will preferably come with aset of pre-configured settings. Exemplary modes may include a grid-videochat mode, a presentation mode, and collaboration mode, which aredescribed herein. Specifying a pre-configured mode may setapplication-dependent priority configuration. Some or a portion of thedefault configuration properties of a pre-configured mode may beoverwritten. A managing account for a communication session canpreferably programmatically set a pre-configured mode prior to or at theinitiation of a new communication session. Alternatively apre-configured mode can be initiated or changed during a communicationsession.

Block S220, which includes resolving media constraints for the set ofinbound media tracks, functions to set media track limits based on mediaproperties or external factors. Media constraints may be inherent mediatrack constraints based on the media format. Media constraints mayadditionally or alternatively be external media constraints, which maylimits supplied by an outside system or entity. For example, anapplication may

Inherent media track constraints are preferably derived or set for eachmedia track based on the properties of the media track. In other words,for the set of inbound media tracks, resolving media constraints caninclude analyzing the media properties for each media track and settingmedia constrains for each media track. Each inbound media track may becommunicated from the source participant in different formats dependingon the client device and/or other factors, which can alter medialimitations per media track. For example, the media format may have aminimal bandwidth threshold (below which media fails or isunsatisfactory) or maximum bandwidth threshold (above which mediaquality does not improve or achieve sufficient improvements).

External media constraints may be set in response toapplication-dependent conditions, a participant client device, or othersignals. In one variation, a managing account may configure mediaconstraints for some or all participants and/or media tracks based ontheir particular use-case.

Resolving media constraints preferably includes determining a minimalbandwidth threshold and maximum bandwidth threshold derived from a mediaformat of a media track. The minimum and/or maximum bandwidth thresholdsmay be based in part on inherent media track constraints and/or externalmedia constraints.

Alternatively, media dimensions, resolution, and/or other mediaproperties could be specified as an indirect approach to indicatingbandwidth settings. Resolving media constraints may additionally includecalculating minimum or maximum bandwidth or other communication factorsfor a media track based on other constraints such as bandwidth usage,but may additionally factor in media properties such as dimensions,bitrate, communication latency, communication jitter, and/or othersuitable properties. In one variation, the used media dimensions of anapplication can be automatically detected and used in specifying themedia bandwidth properties. For example, a library may be configured toautomatically detect media dimensions displayed within an application orbrowser window and then to supply those dimensions. The media dimensionscan be provided during negotiation and setup of media communication.Additionally and/or alternatively, the media dimensions can be deliveredduring a media communication such as when the display of media changesin response to a user interface change.

Block S230, which includes receiving bandwidth estimation feedback andestimating available bandwidth of the outbound link S230, functions touse bandwidth feedback from one or more participants/subscribers.Receiving bandwidth estimation feedback preferably involves a clientdevice transmitting bandwidth or other forms of communicationinformation to the bandwidth management system, and correspondingly, atthe bandwidth management system receiving the bandwidth estimationfeedback. Bandwidth estimation may additionally or alternatively usehistorical or predictive techniques, which can include predictingbandwidth conditions for one or more participants subscribing tooutbound media tracks transmitted in block S300. For example, historicalbandwidth availability feedback collected for an account may be used inestimating available bandwidth in the case where no or only a limitednumber of subscribers report bandwidth usage.

Some variations of the method may not include or use bandwidthestimation or may only use bandwidth estimation for a subset ofparticipants.

Block S240, which includes producing bandwidth allocation configurationbased on at least the media track priority configuration and mediaconstraints, functions to process various objectives, preferences, andlimitations around the media tracks for a given communication link. Thebandwidth allocation configuration preferably characterizes howbandwidth can be allocated when transmitting media tracks in acommunication link. Preferably, block S240 processes track priorities,media constraints, and optionally bandwidth availability estimation. Ingeneral, producing bandwidth allocation configuration functions togenerate an optimized or at least a preferable bandwidth setting for agiven number of media tracks transmitted to a particular participant(e.g., a subscriber to a set of media tracks).

In implementation, block S240 is used in generating individual bandwidthallocation configurations for a set of participants that are subscribersto or destinations of the outbound the media tracks. Accordingly,producing bandwidth allocation configuration can include, for eachparticipant in a set of participant, producing bandwidth allocationconfiguration based on media track priority configuration and the mediaconstraints for each participant. Each participant may have media tracksindividually prioritized and/or particular media constraints. Forexample, two different participants may indicate, through userinteraction on their client device, different media tracks of primaryinterest, which could result in different prioritization of those twomedia tracks. In another example, two different participants may havedifferent hardware/software limitations such that media constraints aredifferent for each participant.

In variations including block S230, where an estimation of availablebandwidth is generated for one or more receiving participants, producinga bandwidth allocation configuration can further be based on availablebandwidth. Accordingly, bandwidth allocation for a set of media tracksin a communication link to a participant will depend on the trackpriority configuration and media constraints of each media track and theestimated available bandwidth of a communication link to theparticipant.

Producing bandwidth allocation configuration preferably includesperiodically or continuously updating the bandwidth allocationconfiguration. Updates may be done in response to changes in priority,constraints, bandwidth estimation, participants, media tracks, and/orany suitable factor.

Preferably, producing bandwidth allocation attempts to maximizebandwidth allocation for bandwidth based on the various factors. Ingeneral, different media tracks will be allocated different bandwidthsettings based on priority, media limitations, and bandwidthavailability. Herein, maximizing and minimizing preferably characterizegeneral increase and decrease of an effective metric. Preferably,maximizing and minimizing includes achieving a local maximum or minimumwithin some window (e.g., seconds, minutes, etc.). However, the use ofdescriptors such as maximum, minimum, or optimize herein does not limitthe method to theoretical metrics and, one could appreciate thatachieving near-maximum or near-minimum or any suitable change thattrends in that direction may have corresponding benefits.

In one preferred implementation, the process of producing bandwidthallocation applies a series of processes that effectively enables anumber of conditions to be considered and used in configuring thebandwidth allocation. In a preferred variation, producing bandwidthallocation maximizes the bandwidth consumed by the tracks. Morespecifically, producing bandwidth allocation configuration can includeapplying a series of processes that comprise a combination of processessuch as: maximizing a sum of bandwidth consumed by the set of mediatracks S2410; verifying the link does not consume more than theavailable bandwidth S2420; verifying that no tracks are over arespective maximum bandwidth S2430; for each track in the set of mediatracks, verifying a track is over a minimum bandwidth S2440; and/orregulating bandwidth allocation such that bandwidth allocation followspriorities of tracks S2450 as shown in FIG. 7.

Block S2410, which includes maximizing sum of bandwidth consumed by allmedia tracks, functions to collectively use the maximum permittedbandwidth allowed across all the media tracks. For example maximizingsum(Bwa_i) where Bwa_i is the bandwidth of a given media track i. Inother words, block S120 takes the maximum value that complies with therest of conditions.

Block S2420, which includes verifying a given link does not consume morethan the available bandwidth functions to enforce the allocatedbandwidth is achievable with the overall bandwidth limitations of thecommunication link. This can include verifying that sum(Bwa_i)<=BWEwhere BWE is the bandwidth estimate for the communication link. BWE maybe based on the output of block S230.

Block S2430, which include verifying that no tracks are over theirrespective maximum, functions to enforce bandwidth maximum constraints.For example, verifying for each media track i in a set of media tracks:Bwa_i<=MaxBw_i.

Block S2440, which includes verifying all tracks are over their minimumor are switched off, functions to enforce bandwidth minimum constraints.For example, verifying for each media track i in a set of media tracks:Bwa_i>=MinBw_i and if not then setting Bwa_i=0 (i.e. switched-off).Other media changes may additionally be used such as changing encoding(e.g., altering compression, format, bitrate, frame rate, resolution,dimensions, etc.) or media type (e.g., converting video to audio only).Verifying that all tracks are over their minimum or are switched offpreferably includes setting the track to a switched off state if thetrack is not over a minimum bandwidth and activating a track in theswitched off state to a switched on state if all the higher prioritytracks have been allocated a specified maximum bandwidth. Such on-offstate evaluation may be evaluated periodically over certain timewindows.

Block S2450, which includes regulating bandwidth allocation such thatbandwidth allocation follows priorities, functions to follow prioritiesof media tracks. Following priorities preferably includes verifying thathigher priority tracks always have higher allocated bandwidth (IfPriority_i>=Priority_j, then Bwa_i>=Bwa_j), lower priority tracks arethe first to be switched off, and the last to be switched on (e.g., ifBwa_i=0, then there is no j so that Priority_j<Priority_i havingBwa_j>0), and non-constrained tracks share bandwidth in proportion to(e.g., a function f) of the priorities. In one variation ofproportionally sharing bandwidth, a track with double priority can beallocated double bandwidth if the function is the identity function. Forexample, assuming that MinBw_i<=Bwa_i<=MaxBw_i and thatMinBw_j<=Bwa_j<=MaxBw_j, then Bwa_i/f(Priority_i)=Bwa_j/f(Priority_j).

In one variation, track bandwidth allocation based on priority mayfollow a monotonic function of their priorities. For example, for theidentity function, that means that tracks with double priority areassigned double bandwidth as long as they are within their constraintbounds. However, other suitable proportioning of bandwidth may be used.

In some variations Block S240 may include generating a learning modelfor bandwidth allocation such that bandwidth allocation configurationmay be set based in part on trained AI/ML learning models.

Block S300, which includes allocating bandwidth to outbound media tracksof a communication link to a client device, functions to apply theassigned bandwidth allocation of block S240 to actual mediacommunication. Allocation of bandwidth can preferably be updated incoordination with changes in bandwidth allocation configuration outputby block S240. As one exemplary situation, different outbound mediatracks will be allocated different amounts of bandwidth for acommunication link to a subscriber. This may be used so that the currentspeaker and optionally other high priority participants are givenproportionally more bandwidth while inactive or lower priorityparticipants have their corresponding media tracks deprioritized.

Allocating bandwidth to outbound media tracks preferably includestransmitting media tracks through the outbound communication link to aparticipant using the allocated bandwidth. If a media track is turnedoff, then the media track may not be transmitted. In some variations, aplaceholder image or asset may be transmitted to serve as a placeholderthough a placeholder may alternatively be handled by the client deviceof the participant. As the method is preferably used in routing media toa plurality of participants, allocating bandwidth to outbound mediatracks can include, for each communication link to a distinctparticipant in a set of communication links, transmitting the set ofmedia tracks through the outbound communication link to the distinctparticipant and, for each media track, using the bandwidth specified inthe bandwidth allocation configuration. Allocating bandwidth to outboundmedia tracks is preferably updated in real-time in response to changesin the bandwidth allocation configuration. Allocating bandwidth tooutbound media tracks may include processing or encoding media tracks ifchanges are necessitated.

In the case of WebRTC, the communication link can be a communicationlink established through an RTCPeerConnection of the WebRTC protocol.Accordingly, allocating bandwidth to outbound media tracks within acommunication link to a client device can include transmitting the mediatracks within a WebRTC peer connection to a client device. That WebRTCpeer connection, as in other forms of RTC communication channels, mayalso be used for receiving uploaded media tracks from the client device.Accordingly, receiving a set of inbound media tracks S100 may includereceiving a media track from the client device through the WebRTC peerconnection. Other suitable synchronous data stream connections mayalternatively be used. In the case of video media tracks, the mediatracks are transmitted as video streams in the communication link to theclient device of the participant. In the case of audio media tracks, themedia tracks are transmitted as audio streams in the communication linkto the client device of the participant.

Architecture

FIG. 8 is a block diagram illustrating an example software architecture906, which may be used in conjunction with various hardwarearchitectures herein described. FIG. 8 is a non-limiting example of asoftware architecture 906 and it will be appreciated that many otherarchitectures may be implemented to facilitate the functionalitydescribed herein. The software architecture 906 may execute on hardwaresuch as machine 1000 of FIG. 9 that includes, among other things,processors 1004, memory 1014, and (input/output) I/O components 1018. Arepresentative hardware layer 952 is illustrated and can represent, forexample, the machine 1000 of FIG. 9. The representative hardware layer952 includes a processing unit 954 having associated executableinstructions 904. Executable instructions 904 represent the executableinstructions of the software architecture 906, including implementationof the methods, components, and so forth described herein. The hardwarelayer 952 also includes memory and/or storage modules 956, which alsohave executable instructions 904. The hardware layer 952 may alsocomprise other hardware 958.

In the example architecture of FIG. 8, the software architecture 906 maybe conceptualized as a stack of layers where each layer providesparticular functionality. For example, the software architecture 906 mayinclude layers such as an operating system 902, libraries 920,frameworks/middleware 918, applications 916, and a presentation layer914. Operationally, the applications 916 and/or other components withinthe layers may invoke application programming interface (API) calls 908through the software stack and receive a response such as messages 912in response to the API calls 908. The layers illustrated arerepresentative in nature and not all software architectures have alllayers. For example, some mobile or special purpose operating systemsmay not provide a frameworks/middleware 918, while others may providesuch a layer. Other software architectures may include additional ordifferent layers.

The operating system 902 may manage hardware resources and providecommon services. The operating system 902 may include, for example, akernel 922, services 924, and drivers 926. The kernel 922 may act as anabstraction layer between the hardware and the other software layers.For example, the kernel 922 may be responsible for memory management,processor management (e.g., scheduling), component management,networking, security settings, and so on. The services 924 may provideother common services for the other software layers. The drivers 926 areresponsible for controlling or interfacing with the underlying hardware.For instance, the drivers 926 include display drivers, camera drivers,Bluetooth® drivers, flash memory drivers, serial communication drivers(e.g., Universal Serial Bus (USB) drivers), Wi-Fi® drivers, audiodrivers, power management drivers, and so forth, depending on thehardware configuration.

The libraries 920 provide a common infrastructure that is used by theapplications 916 and/or other components and/or layers. The libraries920 provide functionality that allows other software components toperform tasks in an easier fashion than to interface directly with theunderlying operating system 902 functionality (e.g., kernel 922,services 924, and/or drivers 926). The libraries 920 may include systemlibraries 944 (e.g., C standard library) that may provide functions suchas memory allocation functions, string manipulation functions,mathematical functions, and the like. In addition, the libraries 920 mayinclude API libraries 946 such as media libraries (e.g., libraries tosupport presentation and manipulation of various media format such asMPEG4, H.264, MP3, AAC, AMR, JPG, PNG), graphics libraries (e.g., anOpenGL framework that may be used to render 2D and 3D in a graphiccontent on a display), database libraries (e.g., SQLite that may providevarious relational database functions), web libraries (e.g., WebKit thatmay provide web browsing functionality), and the like. The libraries 920may also include a wide variety of other libraries 948 to provide manyother APIs to the applications 916 and other softwarecomponents/modules.

The frameworks/middleware 918 (also sometimes referred to as middleware)provide a higher-level common infrastructure that may be used by theapplications 916 and/or other software components/modules. For example,the frameworks/middleware 918 may provide various graphical userinterface (GUI) functions, high-level resource management, high-levellocation services, and so forth. The frameworks/middleware 918 mayprovide a broad spectrum of other APIs that may be used by theapplications 916 and/or other software components/modules, some of whichmay be specific to a particular operating system 902 or platform.

The applications 916 include built-in applications 938 and/orthird-party applications 940. Examples of representative built-inapplications 938 may include, but are not limited to, a contactsapplication, a browser application, a book reader application, alocation application, a media application, a messaging application,and/or a game application. Third-party applications 940 may include anapplication developed using the ANDROID™ or IOS™ software developmentkit (SDK) by an entity other than the vendor of the particular platform,and may be mobile software running on a mobile operating system such asIOS™, ANDROID™, WINDOWS® Phone, or other mobile operating systems. Thethird-party applications 940 may invoke the API calls 908 provided bythe mobile operating system (such as operating system 902) to facilitatefunctionality described herein.

The applications 916 may use built in operating system functions (e.g.,kernel 922, services 924, and/or drivers 926), libraries 920, andframeworks/middleware 918 to create UIs to interact with users of thesystem. Alternatively, or additionally, in some systems, interactionswith a user may occur through a presentation layer, such as presentationlayer 914. In these systems, the application/component “logic” can beseparated from the aspects of the application/component that interactwith a user.

FIG. 9 is a block diagram illustrating components of a machine 1000,according to some example embodiments, able to read instructions 904from a machine-readable medium (e.g., a machine-readable storage medium)and perform any one or more of the methodologies discussed herein.Specifically, FIG. 9 shows a diagrammatic representation of the machine1000 in the example form of a computer system, within which instructions1010 (e.g., software, a program, an application, an applet, an app, orother executable code) for causing the machine 1000 to perform any oneor more of the methodologies discussed herein may be executed. As such,the instructions 1010 may be used to implement modules or componentsdescribed herein. The instructions 1010 transform the general,non-programmed machine 1000 into a particular machine 1000 programmed tocarry out the described and illustrated functions in the mannerdescribed. In alternative embodiments, the machine 1000 operates as astandalone device or may be coupled (e.g., networked) to other machines.In a networked deployment, the machine 1000 may operate in the capacityof a server machine or a client machine in a server-client networkenvironment, or as a peer machine in a peer-to-peer (or distributed)network environment. The machine 1000 may comprise, but not be limitedto, a server computer, a client computer, a PC, a tablet computer, alaptop computer, a netbook, a set-top box (STB), a personal digitalassistant (PDA), an entertainment media system, a cellular telephone, asmart phone, a mobile device, a wearable device (e.g., a smart watch), asmart home device (e.g., a smart appliance), other smart devices, a webappliance, a network router, a network switch, a network bridge, or anymachine 1000 capable of executing the instructions 1010, sequentially orotherwise, that specify actions to be taken by machine 1000. Further,while only a single machine 1000 is illustrated, the term “machine”shall also be taken to include a collection of machines thatindividually or jointly execute the instructions 1010 to perform any oneor more of the methodologies discussed herein.

The machine 1000 may include processors 1004, memory/storage 1006, andI/O components 1018, which may be configured to communicate with eachother such as via a bus 1002. The memory/storage 1006 may include amemory 1014, such as a main memory, or other memory storage, and astorage unit 1016, both accessible to the processors 1004 such as viathe bus 1002. The storage unit 1016 and memory 1014 store theinstructions 1010 embodying any one or more of the methodologies orfunctions described herein. The instructions 1010 may also reside,completely or partially, within the memory 1014, within the storage unit1016, within at least one of the processors 1004 (e.g., within theprocessor's cache memory), or any suitable combination thereof, duringexecution thereof by the machine 1000. Accordingly, the memory 1014, thestorage unit 1016, and the memory of processors 1004 are examples ofmachine-readable media.

The I/O components 1018 may include a wide variety of components toreceive input, provide output, produce output, transmit information,exchange information, capture measurements, and so on. The specific I/Ocomponents 1018 that are included in a particular machine 1000 willdepend on the type of machine. For example, portable machines such asmobile phones will likely include a touch input device or other suchinput mechanisms, while a headless server machine will likely notinclude such a touch input device. It will be appreciated that the I/Ocomponents 1018 may include many other components that are not shown inFIG. 9. The I/O components 1018 are grouped according to functionalitymerely for simplifying the following discussion and the grouping is inno way limiting. In various example embodiments, the I/O components 1018may include output components 1026 and input components 1028. The outputcomponents 1026 may include visual components (e.g., a display such as aplasma display panel (PDP), a light emitting diode (LED) display, aliquid crystal display (LCD), a projector, or a cathode ray tube (CRT)),acoustic components (e.g., speakers), haptic components (e.g., avibratory motor, resistance mechanisms), other signal generators, and soforth. The input components 1028 may include alphanumeric inputcomponents (e.g., a keyboard, a touch screen configured to receivealphanumeric input, a photo-optical keyboard, or other alphanumericinput components), point based input components (e.g., a mouse, atouchpad, a trackball, a joystick, a motion sensor, or other pointinginstrument), tactile input components (e.g., a physical button, a touchscreen that provides location and/or force of touches or touch gestures,or other tactile input components), audio input components (e.g., amicrophone), and the like.

In further example embodiments, the I/O components 1018 may includebiometric components 1030, motion components 1034, environmentalcomponents 1036, or position components 1038 among a wide array of othercomponents. For example, the biometric components 1030 may includecomponents to detect expressions (e.g., hand expressions, facialexpressions, vocal expressions, body gestures, or eye tracking), measurebiosignals (e.g., blood pressure, heart rate, body temperature,perspiration, or brain waves), identify a person (e.g., voiceidentification, retinal identification, facial identification,fingerprint identification, or electroencephalogram basedidentification), and the like. The motion components 1034 may includeacceleration sensor components (e.g., accelerometer), gravitation sensorcomponents, rotation sensor components (e.g., gyroscope), and so forth.The environmental components 1036 may include, for example, illuminationsensor components (e.g., photometer), temperature sensor components(e.g., one or more thermometer that detect ambient temperature),humidity sensor components, pressure sensor components (e.g.,barometer), acoustic sensor components (e.g., one or more microphonesthat detect background noise), proximity sensor components (e.g.,infrared sensors that detect nearby objects), gas sensors (e.g., gasdetection sensors to detect concentrations of hazardous gases for safetyor to measure pollutants in the atmosphere), or other components thatmay provide indications, measurements, or signals corresponding to asurrounding physical environment. The position components 1038 mayinclude location sensor components (e.g., a GPS receiver component),altitude sensor components (e.g., altimeters or barometers that detectair pressure from which altitude may be derived), orientation sensorcomponents (e.g., magnetometers), and the like.

Communication may be implemented using a wide variety of technologies.The I/O components 1018 may include communication components 1040operable to couple the machine 1000 to a network 1032 or devices 1020via coupling 1024 and coupling 1022, respectively. For example, thecommunication components 1040 may include a network interface componentor other suitable device to interface with the network 1032. In furtherexamples, communication components 1040 may include wired communicationcomponents, wireless communication components, cellular communicationcomponents, near field communication (NFC) components, Bluetooth®components (e.g., Bluetooth® Low Energy), Wi-Fi® components, and othercommunication components to provide communication via other modalities.The devices 1020 may be another machine or any of a wide variety ofperipheral devices (e.g., a peripheral device coupled via a USB).

Moreover, the communication components 1040 may detect identifiers orinclude components operable to detect identifiers. For example, thecommunication components 1040 may include radio frequency identification(RFID) tag reader components, NFC smart tag detection components,optical reader components (e.g., an optical sensor to detectone-dimensional bar codes such as Universal Product Code (UPC) bar code,multi-dimensional bar codes such as Quick Response (QR) code, Azteccode, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2Dbar code, and other optical codes), or acoustic detection components(e.g., microphones to identify tagged audio signals). In addition, avariety of information may be derived via the communication components1040 such as location via Internet Protocol (IP) geo-location, locationvia Wi-Fi® signal triangulation, location via detecting a NFC beaconsignal that may indicate a particular location, and so forth.

Glossary

“CARRIER SIGNAL” in this context refers to any intangible medium that iscapable of storing, encoding, or carrying instructions 1010 forexecution by the machine 1000, and includes digital or analogcommunications signals or other intangible medium to facilitatecommunication of such instructions 1010. Instructions 1010 may betransmitted or received over the network 1032 using a transmissionmedium via a network interface device and using any one of a number ofwell-known transfer protocols.

“CLIENT DEVICE” in this context refers to any machine 1000 thatinterfaces to a communications network 1032 to obtain resources from oneor more server systems or other client devices 802, 804 such as shown inFIG. 10. A client device 802, 804 may be, but is not limited to, mobilephones, desktop computers, laptops, PDAs, smart phones, tablets, ultrabooks, netbooks, laptops, multi-processor systems, microprocessor-basedor programmable consumer electronics, game consoles, STBs, or any othercommunication device that a user may use to access a network 1032.

“COMMUNICATIONS NETWORK” in this context refers to one or more portionsof a network 1032 that may be an ad hoc network, an intranet, anextranet, a virtual private network (VPN), a LAN, a wireless LAN (WLAN),a WAN, a wireless WAN (WWAN), a metropolitan area network (MAN), theInternet, a portion of the Internet, a portion of the Public SwitchedTelephone Network (PSTN), a plain old telephone service (POTS) network,a cellular telephone network, a wireless network, a Wi-Fi® network,another type of network, or a combination of two or more such networks.For example, a network 1032 or a portion of a network 1032 may include awireless or cellular network and the coupling may be a Code DivisionMultiple Access (CDMA) connection, a Global System for Mobilecommunications (GSM) connection, or other type of cellular or wirelesscoupling. In this example, the coupling may implement any of a varietyof types of data transfer technology, such as Single Carrier RadioTransmission Technology (1×RTT), Evolution-Data Optimized (EVDO)technology, General Packet Radio Service (GPRS) technology, EnhancedData rates for GSM Evolution (EDGE) technology, third GenerationPartnership Project (3GPP) including 3G, fourth generation wireless (4G)networks, Universal Mobile Telecommunications System (UMTS), High SpeedPacket Access (HSPA), Worldwide Interoperability for Microwave Access(WiMAX), Long Term Evolution (LTE) standard, others defined by variousstandard setting organizations, other long range protocols, or otherdata transfer technology.

“MACHINE-READABLE MEDIUM” in this context refers to a component, deviceor other tangible media able to store instructions 1010 and datatemporarily or permanently and may include, but is not be limited to,random-access memory (RAM), read-only memory (ROM), buffer memory, flashmemory, optical media, magnetic media, cache memory, other types ofstorage (e.g., erasable programmable read-only memory (EEPROM)), and/orany suitable combination thereof. The term “machine-readable medium”should be taken to include a single medium or multiple media (e.g., acentralized or distributed database, or associated caches and servers)able to store instructions 1010. The term “machine-readable medium”shall also be taken to include any medium, or combination of multiplemedia, that is capable of storing instructions 1010 (e.g., code) forexecution by a machine 1000, such that the instructions 1010, whenexecuted by one or more processors 1004 of the machine 1000, cause themachine 1000 to perform any one or more of the methodologies describedherein. Accordingly, a “machine-readable medium” refers to a singlestorage apparatus or device, as well as “cloud-based” storage systems orstorage networks that include multiple storage apparatus or devices. Theterm “machine-readable medium” excludes signals per se.

“COMPONENT” in this context refers to a device, physical entity, orlogic having boundaries defined by function or subroutine calls, branchpoints, APIs, or other technologies that provide for the partitioning ormodularization of particular processing or control functions. Componentsmay be combined via their interfaces with other components to carry outa machine process. A component may be a packaged functional hardwareunit designed for use with other components and a part of a program thatusually performs a particular function of related functions. Componentsmay constitute either software components (e.g., code embodied on amachine-readable medium) or hardware components. A “hardware component”is a tangible unit capable of performing certain operations and may beconfigured or arranged in a certain physical manner. In various exampleembodiments, one or more computer systems (e.g., a standalone computersystem, a client computer system, or a server computer system) or one ormore hardware components of a computer system (e.g., a processor or agroup of processors 1004) may be configured by software (e.g., anapplication 416 or application portion) as a hardware component thatoperates to perform certain operations as described herein. A hardwarecomponent may also be implemented mechanically, electronically, or anysuitable combination thereof. For example, a hardware component mayinclude dedicated circuitry or logic that is permanently configured toperform certain operations. A hardware component may be aspecial-purpose processor, such as a field-programmable gate array(FPGA) or an application specific integrated circuit (ASIC). A hardwarecomponent may also include programmable logic or circuitry that istemporarily configured by software to perform certain operations. Forexample, a hardware component may include software executed by ageneral-purpose processor 1004 or other programmable processor 1004.Once configured by such software, hardware components become specificmachines 1000 (or specific components of a machine 1000) uniquelytailored to perform the configured functions and are no longergeneral-purpose processors 1004. It will be appreciated that thedecision to implement a hardware component mechanically, in dedicatedand permanently configured circuitry, or in temporarily configuredcircuitry (e.g., configured by software), may be driven by cost and timeconsiderations. Accordingly, the phrase “hardware component” (or“hardware-implemented component”) should be understood to encompass atangible entity, be that an entity that is physically constructed,permanently configured (e.g., hardwired), or temporarily configured(e.g., programmed) to operate in a certain manner or to perform certainoperations described herein. Considering embodiments in which hardwarecomponents are temporarily configured (e.g., programmed), each of thehardware components need not be configured or instantiated at any oneinstance in time. For example, where a hardware component comprises ageneral-purpose processor 1004 configured by software to become aspecial-purpose processor, the general-purpose processor 1004 may beconfigured as respectively different special-purpose processors (e.g.,comprising different hardware components) at different times. Softwareaccordingly configures a particular processor or processors 1004, forexample, to constitute a particular hardware component at one instanceof time and to constitute a different hardware component at a differentinstance of time. Hardware components can provide information to, andreceive information from, other hardware components. Accordingly, thedescribed hardware components may be regarded as being communicativelycoupled. Where multiple hardware components exist contemporaneously,communications may be achieved through signal transmission (e.g., overappropriate circuits and buses 1002) between or among two or more of thehardware components. In embodiments in which multiple hardwarecomponents are configured or instantiated at different times,communications between such hardware components may be achieved, forexample, through the storage and retrieval of information in memorystructures to which the multiple hardware components have access. Forexample, one hardware component may perform an operation and store theoutput of that operation in a memory device to which it iscommunicatively coupled. A further hardware component may then, at alater time, access the memory device to retrieve and process the storedoutput. Hardware components may also initiate communications with inputor output devices, and can operate on a resource (e.g., a collection ofinformation). The various operations of example methods described hereinmay be performed, at least partially, by one or more processors 1004that are temporarily configured (e.g., by software) or permanentlyconfigured to perform the relevant operations. Whether temporarily orpermanently configured, such processors 1004 may constituteprocessor-implemented components that operate to perform one or moreoperations or functions described herein. As used herein,“processor-implemented component” refers to a hardware componentimplemented using one or more processors 1004. Similarly, the methodsdescribed herein may be at least partially processor-implemented, with aparticular processor or processors 1004 being an example of hardware.For example, at least some of the operations of a method may beperformed by one or more processors 1004 or processor-implementedcomponents. Moreover, the one or more processors 1004 may also operateto support performance of the relevant operations in a “cloud computing”environment or as a “software as a service” (SaaS). For example, atleast some of the operations may be performed by a group of computers(as examples of machines 1000 including processors 1004), with theseoperations being accessible via a network 1032 (e.g., the Internet) andvia one or more appropriate interfaces (e.g., an API). The performanceof certain of the operations may be distributed among the processors1004, not only residing within a single machine 1000, but deployedacross a number of machines 1000. In some example embodiments, theprocessors 1004 or processor-implemented components may be located in asingle geographic location (e.g., within a home environment, an officeenvironment, or a server farm). In other example embodiments, theprocessors 1004 or processor-implemented components may be distributedacross a number of geographic locations.

“PROCESSOR” in this context refers to any circuit or virtual circuit (aphysical circuit emulated by logic executing on an actual processor1004) that manipulates data values according to control signals (e.g.,“commands,” “op codes,” “machine code,” etc.) and which producescorresponding output signals that are applied to operate a machine 1000.A processor 1004 may be, for example, a central processing unit (CPU), areduced instruction set computing (RISC) processor, a complexinstruction set computing (CISC) processor, a graphics processing unit(GPU), a digital signal processor (DSP), an ASIC, a radio-frequencyintegrated circuit (RFIC) or any combination thereof. A processor 1004may further be a multi-core processor having two or more independentprocessors 1004 (sometimes referred to as “cores”) that may executeinstructions 1010 contemporaneously.

What is claimed is:
 1. A method for bandwidth allocation in multi-trackmedia communication comprising: receiving a set of inbound media tracks;resolving track priority configuration for the set of media tracks;resolving media constraints that are at least partially derived fromproperties of the set of media tracks; producing bandwidth allocationconfiguration based on at least the track priority configuration andmedia constraints, wherein producing bandwidth allocation configurationcomprises applying a series of processes that comprise of at least:maximizing a sum of bandwidth consumed by the set of media tracks,verifying the link does not consume more than the available bandwidth,verifying that no tracks are over a respective maximum bandwidthspecified in the media constraints, for each track in the set of mediatracks, verifying a track is over a minimum bandwidth specified in themedia constraints, and regulating bandwidth allocation such thatbandwidth allocation follows priorities of tracks specified in the trackpriority configuration; and allocating bandwidth to outbound mediatracks within a communication link to a client device wherein bandwidthof the outbound media tracks is allocated based on the bandwidthallocation configuration.
 2. The method of claim 1, further comprisingreceiving bandwidth estimation feedback and estimating availablebandwidth of the communication link; and wherein producing the bandwidthallocation configuration is further based on the available bandwidth. 3.The method of claim 1, wherein the inbound media tracks are video mediatracks from a set of participants; and wherein the communication link tothe client device is a link of video media tracks.
 4. The method ofclaim 1, wherein the inbound media tracks are audio media tracks from aset of participants; and wherein the communication link to the clientdevice is a communication link of audio media tracks.
 5. The method ofclaim 4, further comprising, for each track in the set of media tracks,setting the media track to a switched off state upon determining thatthe media track is not over a minimum bandwidth; and activating a mediatrack in the switched off state to a switched on state upon determiningthat all the higher priority media tracks have been allocated aspecified maximum bandwidth.
 6. The method of claim 1, wherein the trackpriority configuration comprises of media policy configuration selectedfrom a set of modes that comprises of at least a speaker detection mode,a presence detection mode, and a movement detection.
 7. The method ofclaim 1, wherein resolving media constraints comprises determining aminimal bandwidth threshold and maximum bandwidth threshold derived froma media format of a media track.
 8. The method of claim 1, whereinresolving track priority configuration for the set of media trackscomprises: setting media-based track priority, setting use-case trackpriority, and determining the track priority from the media-based trackpriority and the use-case track priority.
 9. The method of claim 1,wherein allocating bandwidth to outbound media tracks within acommunication link to a client device comprises transmitting the mediatracks within a WebRTC peer connection to a client device.
 10. Themethod of claim 9, wherein receiving a set of inbound media trackscomprises of receiving a media track from the client device through theWebRTC peer connection.
 11. A machine-readable storage medium comprisinginstructions that, when executed by one or more processors of a machine,cause the machine to perform operations comprising: receiving a set ofinbound media tracks; receiving track priority configuration for the setof media tracks; resolving media constraints that are at least partiallyderived from properties of the set of media tracks; producing bandwidthallocation configuration based on at least the track priorityconfiguration and media constraints, wherein producing bandwidthallocation configuration comprises applying a series of processes thatcomprise of at least: maximizing a sum of bandwidth consumed by the setof media tracks, verifying the link does not consume more than theavailable bandwidth, verifying that no tracks are over their respectivemaximum bandwidth, for each track in the set of media tracks, verifyinga track is over a minimum bandwidth, and regulating bandwidth allocationsuch that bandwidth allocation follows priorities of tracks; andallocating bandwidth to outbound media tracks within a communicationlink to a client device wherein bandwidth of the outbound media tracksis allocated based on the bandwidth allocation configuration.
 12. Themachine-readable storage medium of claim 11, further comprisinginstructions that cause the machine to perform operations that compriseof receiving bandwidth estimation feedback and estimating availablebandwidth of the outbound link; and wherein producing the bandwidthallocation configuration is further based on the available bandwidth.13. The machine-readable storage medium of claim 11, wherein the inboundmedia tracks are video media tracks from a set of participants; andwherein the communication link to the client device is a communicationlink of video media tracks.
 14. The machine-readable storage medium ofclaim 11, wherein the inbound media tracks are audio media tracks from aset of participants; and wherein the communication link to the clientdevice is a communication link of audio media tracks.
 15. Themachine-readable storage medium of claim 11, further comprisinginstructions that cause the machine to perform operations that compriseof, for each track in the set of media tracks, setting the track to aswitched off state upon determining that the track is not over a minimumbandwidth; and activating a track in the switched off state to aswitched on state upon determining that all the higher priority trackshave been allocated a specified maximum bandwidth.
 16. A system forbandwidth allocation comprising: a media track interface through which aset of inbound media tracks are received; a policy manager configured tospecify a track priority configuration for the set of media tracks; abandwidth management system that is configured to resolve mediaconstraints that are at least partially derived from properties of theset of media tracks, and set bandwidth allocation configuration based onat least the track priority configuration and media constraints; and abandwidth allocator configured to allocate bandwidth to outbound mediatracks within a communication link to a client device wherein bandwidthof the outbound media tracks is allocated based on the bandwidthallocation configuration and wherein the bandwidth allocationconfiguration is set through application of a series of processesconfigured to: maximize a sum of bandwidth consumed by the set of mediatracks, verify the link does not consume more than the availablebandwidth, verify that no tracks are over their respective maximumbandwidth, for each track in the set of media tracks, verify a mediatrack is over a minimum bandwidth and set the track to a switched offstate upon determining that the media track is not over a minimumbandwidth, activate a track in the switched off state to a switched onstate upon determining that all the higher priority tracks have beenallocated a specified maximum bandwidth, and regulate bandwidthallocation such that bandwidth allocation follows priorities of tracks.17. The system of claim 16, wherein the bandwidth management systemfurther comprises a bandwidth estimator configured to estimate availablebandwidth of the outbound link; and wherein configuration to produce thebandwidth allocation configuration is further configured to produce thebandwidth allocation configuration based on the available bandwidth.